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INTRODUCTION 

Information  cues  available  in  laparoscopy  and  other  forms  of  minimally  invasive  surgery 
are  impoverished  relative  to  cues  available  in  open  surgery.  Acquiring  surgical  skill  in 
such  an  environment  is  extremely  challenging.  Even  after  mastery,  continued  practice 
can  lead  to  problems  for  the  surgeon  as  indicated  by  fi'equent  incidence  of  pain  and  injury 
associated  with  laparoscopy.  The  long-term  impact  on  the  surgeon  performing  these 
procedures  is  largely  unknown. 

The  goal  of  this  work  is  to  develop  and  test  new  technologies  that  will  break  down  the 
barriers  that  block  more  surgeons  from  attaining  and  continuing  to  practice  (without 
injury  or  pain)  high  levels  of  skill  in  MIS.  This  project  will  develop  new  technology  by 
concentrating  on  three  major  research  thrusts: 

a.  Smart  Image:  the  project  will  develop  and  evaluate  new  approaches  for 
extracting,  fusing,  and  presenting  information  cues  from  imagery  and  other  data 
sources. 

b.  Configurable  Display:  the  project  will  develop  new  approaches  for  presenting 
existing  data  (video,  CT  data)  and  extracted  cues  (3D  reconstruction,  haptic  cues, 
etc.)  to  the  user  within  a  flexible,  configurable  display  environment 

c.  Ei^onomic  Assessment:  the  project  will  use  existing  technology  and  build  new 
techniques  as  needed  to  acquire  crucial  ergonomic  data  relative  to  key  factors  of 
patient  position,  technology  configuration,  and  instrument  design. 


BODY 

With  the  ^proval  of  the  Grants  Officer,  this  project  has  undergone  substantial  personnel 
changes  in  its  first  year.  These  changes  have  shifted  the  timeline  for  completing  project 
milestones  outlined  in  the  approved  “Statement  of  Work.”  The  new  timeline  for 
completing  proposed  work  is  reflected  in  this  annual  report. 

Specifically,  the  project  has  undergone  the  following  changes  in  the  past  year: 

•  Change  of  PI  (from  Adrian  Park,  M.D.  to  W.  Brent  Seales,  Ph.D.) 

•  Modification  in  Statement  of  Work 

•  Establishment  of  two  new  subcontracts 

The  Principal  Investigator  was  changed  in  response  to  the  change  in  affiliation  of  the 
original  Principal  Investigator,  Adrian  Park,  M.D.,  who  left  ftie  University  of  Kentucky  in 
September  2003  in  order  to  accept  a  position  as  Head  of  General  Surgery  at  the 
University  of  Maryland  Medical  Center  in  Baltimore,  MD.  The  central  role  of 
Information  Technology  in  this  work  made  it  possible  for  W.  Brent  Seales,  a  collaborator 
with  the  original  Principal  Investigator,  to  obtain  approval  from  the  Grants  Officer  to 
assume  the  role  of  Principal  Investigator.  This  change  was  approved  and  executed  in 
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conjunction  with  the  establishment  of  a  subcontract  to  the  University  of  Maryland  to 
continue  progress  on  the  clinical  portion  of  the  project  through  the  work  of  Adrian  Park, 
M.D. 

The  administrative  re-organization  of  the  project  necessitated  a  pause  in  project  effort, 
limiting  progress  toward  project  milestones  for  a  period  of  time.  The  Grants  Officer  has 
been  cognizant  and  supportive  at  all  times  during  this  period  of  reorganization.  In  this 
report  we  show  two  months  of  progress  from  the  original  13-month  schedule.  The  shift  is 
the  result  of  protracted  negotiations  leading  to  the  relocation  of  Adrian  Park,  M.D.  to 
Maryland,  together  with  the  administrative  overhead  of  the  change  of  principal 
investigator  and  the  time  cost  of  moving  and  re-establishing  the  development  laboratory 
in  a  new  location. 

In  the  following  we  report  progress  relative  to  Information  Technology  milestones  (a) 
through  (e)  from  the  approved  “Statement  of  Work:” 

Primary  Milestones:  IT 

a.  Hire  and  train  development  personnel,  (months  1-4) 

b.  Establish  software  development  environment,  (hardware,  software,  standards) 
(months  2-4) 

c.  Establish  MIS  test-bed.  (collect  instruments,  interfaces)  (months  4-6) 

d.  Gather  product  requirements,  (months  1-4) 

e.  Analyze  product  requirements,  (months  2-5) 

f.  Hold  a  multi-site  project  workshop  meeting,  (month  10) 

g.  Define  a  scalable  real-time  architecture  for  multi-input,  multi-transformation, 
multi-display  MIS  support,  (months  4-6) 

h.  Identify  re-usable  components  from  public  domain  and  outside  researchers, 
(months  6-8) 

i.  Implement  &st  iteration  of  architectural  firework,  (months  7-12) 

j.  Implement  flexible  display  back-end  for  unprocessed  probe  camera  data,  (months 
5-10) 

We  have  hired  two  full-time  developers,  each  with  M.S.  degrees  in  computer  science 
specialized  in  the  area  of  computer  vision.  These  developers  have  established  a  software 
and  hardware  development  environment  according  to  current  industry  standards  under  the 
direct  supervision  of  Dimcan  Clarke,  Ph.D.  (project  technical  lead  and  software 
engineering  expert).  The  two  full-time  developers  have  systematically  gathered 
requirements  and  have  worked  with  project  leaders  to  configure  the  working  environment 
for  an  MIS  test-bed.  This  process  has  been  accelerated  through  the  help  of  Adrian  Park, 
M.D.  and  his  associates  at  Stryker  Endoscopy,  who  have  provided  MIS  instruments  and 
consultation  as  needed. 

The  initial  progress  toward  completion  of  milestones  (a)-(e)  is  reflected  by  the  successful 
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publication  of  an  abstract  at  MMVR  2004,  included  in  the  appendix  of  this  report. 

We  are  able  to  report  progress  relative  to  Ergonomic  Assessment  milestones  (a)  through 
(c)  from  the  approved  “Statement  of  Work:” 

Primary  Milestones:  Ergonomic  Assessment 

a.  Obtain  UMD IRB/IACUC  approval,  (months  0-2) 

b.  Hire  and  train  development  persoimel.  (months  1-4) 

c.  Establish  ergonomic  assessment  environment  (space,  hardware,  software, 
assessment  methodologies),  (monihs  1-4) 

d.  Organize  hiunan  subjects  for  baseline  tasks,  (months  2-4) 

e.  Analyze  assessment  hardware/software  system  requirements,  (months  2-5) 

f.  Perform  baseline  assessment  study  using  human  subjects  completing  tasks  in 
training  environment,  (months  4-6) 

g.  Define  technical  requirements  for  improved  assessment  technology,  (months  4-6) 

h.  Process  and  interpret  baseline  study  using  trainer  boxes  and  assessment 
environment,  (months  6-8) 

i.  Hold  a  multi-site  project  workshop  meeting,  (month  10) 

j.  Conduct  follow-on  study  with  humans  completing  laparoscopic  baseline  tasks  on 
animals,  (months  8-10) 

Ergonomic  assessment  milestones  are  to  be  completed  at  the  University  of  Maryland 
under  a  subcontract  requiring  new  IRB  approval.  (This  is  beyond  the  original  approval 
which  was  secured  at  the  University  of  Kentucky.)  We  have  obtained  University  of 
Maryland  IRB  approval  and  are  completing  the  lACUC  process  now.  We  are  in  the  first 
phase  of  hiring  personnel  (b)  and  securing  space  at  University  of  Maryland  (c)  for 
hardware  setup  and  evaluation. 

We  do  not  anticipate  substantial  deviation  from  the  approved  “Statement  of  Work”  other 
than  the  stated  time  shift  due  to  project  reorganization. 

KEY  RESEARCH  ACCOMPLISHMENTS 

At  this  point  our  key  research  accomplishments  primarily  involve  the  establishment  of  a 
stable  first-rate  working  environment  from  which  to  perform  the  work: 

•  Recruitment  and  training  of  two  full-time  software  developers  with  M.S.  degrees 
in  Computer  Science  and  detailed  domain  knowledge  in  computer  vision. 

•  Set  up  of  commercial-grade  software/hardware  development  environment. 

•  Renovation  and  configuration  of  MIS  experimental/simulation  test  bed. 

•  Initial  set  of  requirements  specifications  developed  for  upcoming  implementation 
and  development  cycle. 

•  Publication  and  presentation  of  abstract  in  MMVR  2004. 

•  Securing  IRB  approval  at  University  of  Maryland. 
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REPORTABLE  OUTCOMES 

•  Manuscript:  MMVR  2004. 

•  Manuscript:  Seminars  in  Laparoscopic  Surgery. 

•  Electronic  project  software  archive  conforming  to  commercial  software 
principles. 

•  Electronic  archive  of  project  progress  and  white-paper  development. 
CONCLUSIONS 

Work  is  currently  on  schedule  relative  to  our  current  “Statement  of  Work”  and  the 
revised  schedule  that  commenced  in  early  December  with  receipt  of  final  approval  and 
authorization  to  resume  work.  We  anticipate  successful  completion  of  proposed  work 
without  substantial  deviations  or  a  reduction  in  scope.  We  will  produce  an  intermediate 
report  of  “so  what”  conclusions  as  soon  as  more  substantive  progress  has  been  made 
toward  project  deliverables. 

REFERENCES 

•  Seales,  W.B.  and  Caban,  J.  “Reconstruction  and  Enhancement  in  Monocular 
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APPENDICES 
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•  Seales,  W.B.  and  Caban,  J.  “Reconstruction  and  Enhancement  in  Monocular 
L^aroscopic  Imageiy”,  Medicine  Meets  Virtual  Reality  (MMVR  2004), 
Newport  Beach,  California. 

•  Seales,  W.B.  and  Caban,  J.  “Visualization  Trends:  Applications  in 
Laparoscopy”,  Seminars  in  Laparoscopic  Surgery.  October  2003. 
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Reconstruction  and  Enhancement  in  Monocular  Laparoscopic  Imagery 


Jesus  J.  Caban  * 

W.  Brent  Seales 
Computer  Science  Department 
University  of  Kentucky 


Abstract 

Constrained  minimally-invasive  surgical  environments  create  a  number  of  challenges for  the  surgeon 
and  for  automated  tools  designed  to  aid  in  the  performance  and  analysis  of  complex  procedures.  The 
3D  reconstruction  of  the  operative  field  opens  up  a  number  of possibilities  for  immersive  presentation, 
automated  analysis,  and  post-operative  evaluation  of  surgical  procedures. 

This  paper  presents  a  method  for  estimating  complete  3D  information  about  scope  and  instrument 
positioning  from  monocular  imagery.  These  measurements  can  be  used  as  the  basis  for  deriving  and 
presenting  additional  cues  during  procedures,  and  can  also  be  used for  post-procedure  analysis  such  as 
objective  estimates  of  high-level  performance  measures  like  economy  of  motion  and  ergonomic  metrics. 


1  Introduction 

Minimally-invasive  surgery  (MIS)  provides  a  number  of  benefits  to  the  patient,  including  lower  risk  of  infection  and 
swifter  recovery  times.  Minimal  invasion  is  accomplished  primarily  through  the  use  of  a  camera  (endoscope)  and  other 
surgical  instruments  inserted  into  the  abdominal  cavity  through  small  “keyhole”  incisions.  The  surgeon  navigates  the 
process  by  viewing  imagery  on  an  external  display.  By  moving  the  endoscope  and  instruments,  complex  surgical  tasks 
can  be  efliciently  accomplished. 

In  this  work  we  present  a  method  to  extract  and  make  explicit  information  that  is  implicitly  confounded  in  the 
imagery.  Such  information,  though  valuable  as  a  direct  cue,  is  usually  subtle,  especially  in  monocular  imagery.  Ex¬ 
tracting  an  explicit  representation  can  provide  a  ready  cue  or  an  analytical  tool  that  otherwise  would  remain  subtle 
and  far  less  useful.  In  particular,  we  concentrate  on  the  problem  of  recovering  the  3D  position  and  orientation  of 
instruments  within  the  endoscope’s  view,  as  well  as  the  distance  of  these  instruments  from  the  scope,  from  each  other, 
and  from  the  anatomy. 

Providing  3D  information  is  crucial  in  addressing  one  of  the  primary  technical  and  visual  obstacles  in  conducting 
MIS  procedures,  which  is  the  lack  of  an  explicit  depth  cue.  Experts  become  very  good  at  understanding  3D  rela¬ 
tionships  from  monocular  imagery,  which  does  not  make  depth  explicit  but  does  contain  a  number  of  subtle  depth 
cues,  such  as  perspective  distortion  and  scale,  expert  knowledge  of  instrument  size,  shape  and  relative  positioning,  and 
narrow  depth  of  field  which  provides  a  focus  cue. 

We  believe  that  the  ability  to  extract  precise  depth  measurements,  including  the  position  and  orientation  of  in¬ 
struments,  scopes  and  anatomy,  can  substantially  enhance  laparoscopic  environments  of  the  future.  In  particular,  we 
envision  two  immediate  uses  when  depth  information  can  be  made  explicit  for  tracked  instruments  and  anatomy:  en¬ 
hanced  visualization  for  the  surgical  team,  and  objective  performance  measures  given  video  of  training  and  simulation 
cases. 

In  the  case  of  enhanced  visualization,  depth  cues  extracted  from  monocular  imagery  can  be  used  to  provide  alter¬ 
native  views  of  instrument  position,  for  example.  Top  views,  closing  distance,  velocity  and  accelerations  between  two 
instruments,  and  real  metric  measurements  within  the  operative  field  to  give  an  accurate  sense  of  the  scale  of  what  is 
potentially  a  very  small  operating  space  are  all  possible  when  depth  can  be  extracted  from  the  imagery. 

With  respect  to  performance  measures,  we  can  formulate  motion  signatures  of  instruments  as  a  function  of  time, 
incorporating  distance,  orientation,  and  the  derivatives  of  these  values  over  time.  We  believe  such  measures  can  give 
a  precision  to  the  problem  of  analyzing  the  performance  of  a  task  completed  within  a  simulation  environment  or  a 
training  box. 

We  believe  that  both  enhanced  visualization  and  objective  performance  metrics  are  valuable.  They  rely  on  the 
extraction  of  metric  3D  information.  Stereo  scope  systems,  such  as  the  da  Vinci  System  [1],  use  a  stereo  endoscope  to 
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Figure  1:  (left)  Original  image  obtained  from  a  35°  endoscope,  (center)  Image  after  undistortion,  (right)  Stapler 
instrument  with  identifiable  points  and  known  measurements. 


make  depth  explicit  for  a  viewer  using  a  corresponding  stereo  viewing  system.  These  systems  can  also  benefit  from 
enhancements  and  metrics  based  on  depth,  which  is  made  even  more  accessible  through  the  introduction  of  the  stereo 
scope.  One  of  the  crucial  contributions  of  this  work  is  that  even  when  a  stereoscopic  scope  is  not  available,  we  can 
obtain  certain  metric  depth  measurements.  This  allows  us  to  apply  enhancements  and  metrics  to  environments  such  as 
trainer  boxes,  archival  procedure  video,  and  current  operating  areas  where  it  is  not  practical  or  even  desirable  to  use 
specialized  hardware  like  the  Da  Vinci  stereo  scope/display. 

In  the  following  sections  we  show  how  to  model  and  calibrate  the  camera,  and  we  outline  the  cue  extraction 
process.  We  follow  this  with  results  from  real  video  that  indicate  the  value  of  metrics  based  on  extracted  cues  as  we 
have  formulated  them.  Related  work  for  visual  tracking  of  laparoscopic  surgery  [3, 2]  shows  that  there  continues  to  be 
a  need  for  new  enhancement  techniques,  metrics  for  analysis,  and  exploration  of  how  best  to  present  extracted  cues  to 
the  surgical  team. 

2  The  Calibrated  Endoscope 

In  order  to  formally  model  the  geometry  of  the  endoscope,  we  assume  that  the  imaging  system  can  be  modeled  as 
a  pinhole  system  (i.e.,  perspective  projection).  Using  this  camera  model,  we  apply  computer  vision  methods  and 
algorithms  in  order  to  calculate  its  characteristic  geometry  and  distortion  parameters. 

First,  we  compute  endoscope  parameters  and  characteristics  through  a  calibration  process.  According  to  the  pinhole 
model,  the  relationship  between  a  3D  point  M  and  its  2D  image  projection  m  is  given  by  sm  ^  A[Rt]M  with 

uo  \ 

VO  1  (1) 

where  5  is  an  arbitrary  scale  factor,  {R,  t)  are  the  extrinsic  parameters,  and  A  is  a  representation  of  the  intrinsic 
parameters.  Extrinsic  parameters  locate  the  camera  in  a  3D  world  coordinate  frame;  intrinsic  parameters  describe 
internal  camera  features  such  as  the  pixel  coordinates  of  the  image  center  and  the  focal  length.  More  specifically,  A  is 
a  3x3  matrix  that  relates  the  pixel  coordinate  system  to  the  world  coordinate  system.  Contained  in  A  are  parameters 
au  and  ay,  which  together  represent  the  focal  length  and  express  the  total  magnification  of  the  imaging  system  that 
results  from  both  optics  and  image  sampling.  Also  contained  in  A  are  the  parameters  uq  and  vq,  which  represent  the 
pixel  coordinates  of  the  image  center. 

As  a  result  of  the  radial  curvature  of  camera  lens  elements,  there  is  no  real  lens  system  that  can  produce  perfect 
pinhole  images.  In  the  case  of  endoscopes,  different  viewing  angle  scopes,  which  enlarge  the  field  of  view,  and 
scopes  with  wide-angle  lenses,  cause  significant  distortion.  These  distortions  can  be  removed  by  calculating  distortion 
parameters  through  optical  calibration.  After  a  camera  has  been  calibrated,  it  is  possible  to  use  the  camera  parameters 
to  resample  any  image  taken  by  that  camera  so  that  its  lens  distortion  is  removed  from  the  image.  Figure  1  (left 
and  center)  shows  lens  distortion  in  an  image  captured  from  a  35°  endoscope  compared  with  a  distortion-free  image 
generated  after  calibration. 

The  key  assumption  that  enables  depth  reconstruction  of  instruments  visible  in  monocular  sequences  is  knowledge 
of  the  shape,  size  and  the  metric  measurements  of  visually  identifiable  fiducials  on  the  instrument.  Based  on  this 
information,  it  is  possible  to  track  features  in  imagery  and  recover  the  3D  position  of  each  tracked  point.  From  these 
points,  with  a  priori  information  about  the  instrument,  it  is  possible  to  compute  the  3D  position  and  orientation  of  the 
tip  of  the  instrument.  Figure  1  (right)  shows  a  stapler  instrument  with  identifiable  marks  and  known  distances  between 
each  of  the  points. 
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Figure  2:  (left)  The  widely-varying  curve  shows  novice  performance;  smoother  motion  is  achievable  by  the  expert, 
(right)  The  linear  constraint  leads  to  a  depth  solution  for  known  points  in  monocular  imagery. 


3  Reconstruction  Method 

The  shape  of  MIS  instruments  is  almost  always  linear  so  that  the  instrument  can  be  smoothly  inserted  into  ports 
and  manipulated  through  small  incisions.  We  exploit  this  fact  and  as  a  result  simplify  the  problem  of  tracking  and 
estimating  3D  points  that  lie  on  the  instrument.  By  selecting  R  =  I  and  t  =  0  in  Equation  1,  we  can  compute  that 
A  =  ZAA^^a,  B  —  zbA~H,  and  C  =  zcA~^c,  where  za^zb  and  zc  are  the  unknown  depths  of  the  A,  B,  C 
points.  These  equations  and  constraints  lead  to  a  solution  [4]  for  unknown  depth  wherever  the  instrument  appears 
in  the  imagery.  These  depth  values  are  derived  based  on  the  assumption  that  the  instruments  are  linear,  the  camera 
is  calibrated  (i.e.,  the  projection  matrix  is  available  from  the  off-line  calibration  process),  and  the  distances  between 
points  on  the  instrument  are  known  a  priori. 

4  Results  and  Conclusion 

We  have  calibrated  various  endoscopes  (e.g.,  0°  and  35°  lenses).  After  calibration,  we  used  recovered  lens  distortion 
estimates  to  remove  lens  distortion  from  images.  Using  the  distortion-corrected  images,  we  tracked  the  shaft  of  a 
stapler  instrument  (Fig.  2  (right))  in  order  to  recover  estimates  of  the  3D  coordinates  of  points  on  the  instrument.  We 
have  found  these  methods  to  be  very  promising  as  a  way  to  recover  3D  cues  from  monocular  data. 

As  an  example,  consider  the  graph  in  Fig.  2  (left).  The  two  curves  on  this  graph  show  3D  motion  estimates  for 
the  stapler  instrument  over  a  set  of  frames.  The  value  plotted  as  the  height  of  the  curve  for  each  frame  value  is  the 
3D  position  of  the  instrument  measured  relative  to  a  fixed  point.  The  curve  that  corresponds  to  the  expert  performing 
the  stapling  action  shows  much  less  relative-motion  variation  than  the  curve  corresponding  to  the  novice.  In  this  case, 
economy  of  motion  over  a  set  of  frames,  evaluated  in  3D  to  capture  movement  toward  and  away  from  the  camera, 
shows  how  an  expert  handles  the  instrument  in  a  way  that  is  measurably  and  objectively  different  from  the  novice. 

A  number  of  interesting  cues  that  exist  in  a  subtle,  implicit  way  in  monocular  laparoscopic  imagery  can  be  recovered 
automatically,  enhanced  and  made  explicit.  This  work  shows  an  example  based  on  the  recovery  of  3D  information 
relative  to  the  frame  of  the  camera  system  from  points  detected  on  visible  instruments  as  well  as  points  in  the  operative 
field  tracked  over  time.  These  methods  do  not  assume  a  redesigned  camera  system,  and  therefore  can  be  of  value  for 
experts  using  and  training  with  monocular  systems  today. 
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Recent  advances  in  visualization  technology  are  being  driven 
by  two  important  trends:  (1)  continued  increases  in  speed  and 
function  of  hardware  devices  and  (2)  increasingly  parallel,  dis¬ 
tributed,  cooperative  systems.  The  incorporation  of  fast,  pow¬ 
erful  devices  into  cooperative  systems  enables  a  complex  in¬ 
terplay  of  sensors,  displays,  and  computational  components 
that  can  create  a  seamless,  perceptually  rich  and  flexible  en¬ 
vironment.  Although  these  trends  have  fueled  a  number  of  ad¬ 
vances  in  visualization  research,  the  unique  requirements  of 
laparoscopy  make  direct,  effective  use  of  visualization  tech¬ 
nology  as  it  Is  applied  in  other  contexts  extremely  challeng¬ 
ing.  This  article  discusses  promising  new  capabilities  in  visu¬ 
alization  technology.  The  costs  and  tradeoffs  create  new  chal¬ 
lenges,  which  are  addressed  In  some  visualization  applica¬ 
tions,  but  must  be  carefully  assessed  in  the  context  of  the  la¬ 
paroscopic  environment  Incorporating  new  visualization  tech¬ 
nology  in  a  way  that  captures  its  benefits  and  meets  stringent 
laparoscopic  requirements  will  very  likely  precipitate  an  enor¬ 
mous  surge  forward  in  the  capabilities  of  the  surgical  team 
and  in  the  quality  of  patient  care. 

The  laparoscopic  environment  is  intentionally 
constrained  in  order  to  benefit  the  patient. 
Minimizing  the  damage  in  establishing  access  to  the 
operative  field  greatly  improves  recovery  times  and 
lessens  the  risk  of  postoperative  complications. 
Although  these  constraints  benefit  the  patient,  they 
create  a  physical  and  informational  bottleneck  that 
spawns  a  number  of  serious  technical  challenges. 
For  the  patient,  "less  is  more" — ^less  invasive  access 
is  more  beneficial.  For  the  surgical  team,  however, 
"less  is  more"  means  that  less  invasive  access  leads 
to  procedures  that  are  more  challenging,  require 
more  skill,  more  training,  and  reliance  on  more 
technologically  sophisticated  tools.  This  article  fo¬ 
cuses  on  how  current  trends  in  visualization  tech¬ 
nology  may  be  applied  in  laparoscopy  to  put  more 
and  more  of  the  burden  on  the  technology,  enabling 
the  surgeon  to  give  the  patient  more  benefit  with 
less  risk. 

Substantial  technical  development  has  es¬ 
tablished  an  environment  that  is  both  minimally 
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invasive  and  procedurally  viable.  Although  this 
environment  is  now  widely  used  for  a  variety  of  pro¬ 
cedures,  the  construction  of  its  interface  lags  far  be¬ 
hind  current  visualization  technology.  Newly  devel¬ 
oped  visualization  technology  moves  beyond  the 
constraints  of  the  windows,  icons,  menus,  and 
pointing  devices  (WIMP)  interface  and  its  associat¬ 
ed  computer  engine  (the  tightly-coupled,  single 
CPU  computer). ^’2  application  to  laparoscopy, 
this  can  create  potential  advances  by  breaking  bar¬ 
riers  that  may  seem  now  to  be  inherent  and  un¬ 
avoidable.  For  example,  it  may  seem  that  the  best 
laparoscopic  view  of  the  operative  field  must  inher¬ 
ently  be  a  two-dimensional  (2D)  image  sequence, 
captured  by  a  camera  and  delivered  to  the  surgeon 
on  a  2D  computer  monitor.  As  other  work  has 
shown,  however,  this  2D  assumption  can  and  is 
being  challenged.^ 

Similarly,  it  may  seem  that  preoperative  data, 
such  as  computed  tomographic  (CT)  scans  and 
magnetic  resonance  imaging  (MRI)  cannot  play  a 
role  in  the  constrained,  minimally  invasive  surgical 
(MIS)  environment  because  of  die  disparities  and 
barriers  involved:  preoperative  data  may  not  be 
available  in  the  operating  room,  3D  fixed-scan  data 
cannot  be  adequately  aligned  with  2D  live-camera 
views,  and  the  surgical  team  cannot  effectively  con¬ 
trol  how  to  optimize,  navigate,  and  control  the 
many  possible  ways  to  fuse  different  data.  The  ques¬ 
tion  is  "can  new  visualization  technologies  continue 
to  expand  the  limits?" 

Of  interest  is  the  similarity  between  the  oper¬ 
ating  room  as  a  whole,  with  its  sterile  boundary 
that  divides  and  separates  it  from  the  outside  con¬ 
text,  and  the  minimally  invasive  environment  that 
has  its  own  set  of  boundaries  and  constraints.  Set 
within  the  larger  context  of  the  operating  room, 
minimally  invasive  procedures  establish  a  second 
boundary  that  divides  and  separates  the  surgeon  in 
the  operating  room  from  the  operative  field.  The 
same  challenges  in  surmounting  the  required  bar¬ 
riers  of  the  operating  room  apply  to  the  embedded 
environment  of  the  MIS  operative  field.  It  is  very 
likely  that  the  same  technologies  will  be  the  basis 
for  solutions. 

What  elements  of  visualization  environments  are 
emerging  with  current  trends?  The  crucial  compo¬ 
nents  are  acquisition,  modeling  and  processing,  and 
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display.  In  the  current  laparoscopic  environment,  the 
camera  is  a  sensor  for  image  acquisition,  the  mod¬ 
eling  and  processing  step  is  embedded  within  the 
camera  hardware,  and  a  cathode  ray  tube  (OIT)  is 
the  display  device.  As  shown  in  Figure  1,  the  scope 
is  tightly  coupled  with  the  display,  and  the  surgical 
team  must  integrate  information  between  and 
among  devices  as  well  as  the  patient.  It  can  require 
substantial  effort  for  the  surgeon  to  integrate  and 
correlate  external  views  of  the  patient  (Figure  1, 
right)  with  simultaneous  intemd  views  visible  on 
the  display. 

Within  the  operating  room,  there  is  very  litde 
notion  of  computation  for  the  purpose  of  cue  de¬ 
tection,  enhancement,  modeling,  and  integration — 
all  devices  function  separately.  It  is  the  surgical 
team's  responsibility  to  interpret  the  imagery  and 
continuously  evaluate  the  procedure  by  integrating 
and  understanding  information  from  a  parallel  set 
of  completely  separate  devices.'* 

Current  visualization  environments  address 
this  problem  by  going  beyond  the  WIMP 
metaphor  and  grappling  with  the  issues  that  are 
required  to  support  immersion,  interaction,  seal- 
ability,  and  data  integration  from  diverse 
sources.  The  requirements  of  the  application  are 
crucial,  since  they  dictate  what  can  be  traded  in 
order  to  develop  a  workable  solution  for  the 
problem  at  hand.  The  point  is  to  identify  trends 
in  visualization,  including  tradeoffs  and  task- 
based  requirements,  and  consider  how  to  appro¬ 
priately  exploit  the  technology  for  improvements 
in  laparoscopy. 


It  is  important  to  note  the  clear  distinction  be¬ 
tween  the  operating  room  environment  and  prac¬ 
tice-oriented  training  and  simulation  environments. 
Training  environments  continue  to  advance  at  a 
rapid  rate  and  provide  a  safe  test  bed  for  developing 
new  ideas.®"®  The  requirements  are  much  higher  in 
the  operating  room,  however.  The  goal  is  to  deploy 
new,  revolutionary  technology  in  the  operating 
room  and  to  develop  a  simulation  environment  that 
so  closely  resembles  smgical  procedures  that  it  is 
indistinguishable.  Toward  that  goal,  simulation  and 
training  tests,  especially  for  visualization  technolo¬ 
gy,  are  an  extremely  useful  strategy  for  assessing 
the  potential  of  a  new  technology.  Success  in  simu¬ 
lation  is  a  precursor  to  success  at  bringing  new 
technology  to  the  operating  room. 

It  is  also  important  to  observe  that  visualiza¬ 
tion  is  not  the  same  thing  as  virtual  reality.  In  fact, 
equating  these  two  may  unnecessarily  bias  solu¬ 
tions  and  approaches  toward  aspects  of  visualiza¬ 
tion  that  are  highlighted  in  virtual  reality  systems, 
such  as  "spatial  immersion."  What  is  necessary  is  a 
careful  look  at  the  requirements  and  the  structure 
of  the  current  laparoscopic  environment,  with  the 
continued  goal  of  improving  it.  The  cognitive  bur¬ 
den  is  already  great  within  the  walls  of  the  operat¬ 
ing  room;  it  is  intensified  by  the  added  constraint 
of  minimal  invasion.  Successful  visualization  tech¬ 
nology  for  the  laparoscopist  must  lessen  that  bur¬ 
den  and  yet  increase  the  ability  to  form  and  exe¬ 
cute  a  complex,  successful  procedure  under  heavy 
constraints. 


Figure  1;  The  laparoscopic  environment:  the  scope,  instru¬ 
ments  and  trocars  are  shown  in  a  simulation  setup  and  are  very 
tightly  coupled,  which  minimizes  latency.  The  arngiced  team 
must  often  integrate  and  correlate  information  while  looking 
directly  at  the  patient  and  simultaneously  observing  a  number  of 
separate  devices. 
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Visualization  Technology 

Visualization  is  the  human  activity  of  exploring,  in¬ 
terpreting,  and  understanding  complex  data.  The 
computational  aspect  of  visualization  is  a  transfor¬ 
mation  of  symbols:  turning  numbers  and  raw  data 
into  geometry,  which  enables  the  human  observer 
to  interpret  complex  simulations  and  phenomena.^ 
The  data  that  form  the  basis  of  this  transformation 
pipeline  must  be  acquired,  processed,  and  displayed. 
Technology  is  criticd  at  each  of  these  points. 

Accepted  laparoscopic  surgical  environments 
rely  on  imagery  acquired  with  a  camera  system 
that  illuminates  and  images  the  region  of  inter- 
est.®>9  After  acquisition,  the  imagery  is  processed 
and  manipulated  in  order  to  improve  its  presenta¬ 
tion  to  the  viewer.  The  imagery  is  then  presented 
on  a  display  system,  most  often  a  CRT  or  flat  panel 
display,  which  is  positioned  within  easy  view  of 
the  surgical  team. 

Advances  in  visualization  environments  derive 
from  improvements  in  data  acquisition,  modeling 
and  processing,  and  display,  and  in  the  overall  ar¬ 
chitecture  to  support  the  interplay  between  these 
elements.  The  development  of  new  types  of  acqui¬ 
sition  technology  must  be  supported  by  expanded 
algorithms  and  architectures  for  manipulating  and 
structuring  the  acquired  data  as  well  as  new  dis¬ 
play  technology  that  efficiently  communicates  this 
structure  to  the  observer.  As  a  system,  the  infor¬ 
mation  bandwidth  is  increasing  at  the  sensor  and  at 
the  display,  supported  by  more  processing  power 
in  between. 


Sensors  and  Acquisition 

Both  sensors  and  algorithms  (simulations)  can  pro¬ 
duce  data  for  visualization.  Sensors  function  at  the 
point  of  contact  with  a  physical  phenomenon  in 
order  to  sample  its  essential  characteristics. 
Simulation  data  are  obtained  from  an  algorithm. 
Often  a  visualization  environment,  such  as  weather 
simulation,  teleimmersion,  telecollaboration,  and 
distance  learning,  combines  real  and  simulated  data. 

The  continued  trend  of  exponential  speedup  in 
computingi^'^2  and  device  miniaturization  directly 
affects  both  real  data  acquisition  and  the  produc¬ 
tion  of  simulation  data.  Smaller  devices  equipped 
with  more  processing  power  can  detect  and  deliver 
better  data,  such  as  higher-resolution  cameras-on-a- 
chip  and  3D  sensors.  Miniaturization  is  certain¬ 
ly  a  welcome  trend  for  laparoscopy,  where  smaller 
sensors  support  the  goal  of  minimd  invasion.  Every 


indication  is  that  devices  will  continue  to  get  small¬ 
er  and  more  powerful.  For  example,  the  Program¬ 
mable  Digital  Camera  project^^  investigates  algo¬ 
rithms,  circuits,  and  different  architectures  for 
portable,  small,  flexible,  and  high-resolution  digital 
camera  sensors. 

As  individual  sensors  increase  in  resolution  and 
decrease  in  size,  an  equally  profoimd  trend  is  that 
many  more  sensors  are  being  used  cooperatively 
within  a  single  system.  Rather  than  relying  on  one, 
super-high-resolution  camera,  for  example,  systems 
use  a  number  of  distributed,  lower-resolution  cam¬ 
eras.  The  Office  of  the  Future  environment  uses 
networks  of  cameras  and  microphones  to  capture 
and  then  remotely  render  images  and  audio.^®  That 
visualization  environment  is  constructed  as  a  way 
to  promote  immersive  collaboration,  and  it  does  so 
through  cooperative,  distributed  sensors  (cameras 
and  microphones)  in  order  to  create  a  sense  of 
presence  between  physically  separate  sites. 

The  use  of  many  sensors  increases  the  com¬ 
putational  load  and  creates  the  problem  of  data 
fusion.  Systems  have  found  solutions  by  coupling 
the  distributed  sensing  network  with  a  distributed 
computing  architecture  that  can  process  data  in 
parallel  and  fuse  data  through  distributed  algo¬ 
rithms  that  communicate  via  high-speed  network 
connectivity.  As  shown  in  the  architectural  dia¬ 
gram  of  Figure  2,  clusters  of  machines  connected 
via  a  high-speed  network  manage  tasks  such  as 
device  integration  and  control,  complex  simula¬ 
tion  processing,  and  display.  This  departure  from 
the  "single  device"  architecture  (Figure  2,  left) 
supports  scalability,  parallelism,  and  information 
fusion. 

Large-scale  visualization  environments  almost 
all  rely  on  clusters  of  machines  to  perform  render¬ 
ing  and  to  drive  large-scale  simulations.^^-^^  Within 
surgical  simulation,  the  Virtual  Reality  Based 
Laparoscopic  Smrgery  Simulator  project^®  uses  an 
ar^itecture  based  on  a  parallel  processing  engine 
to  support  the  communications,  computation,  and 
processing  needed  for  laparoscopic  applications. 
Similarly,  the  daVinci  system^^  implements  a  tight¬ 
ly  coupled  set  of  processes  to  control  servos  and 
manage  a  user  interface. 

Distributed  miniature  sensors  that  are  driven 
by  a  distributed  computing  environment  are 
botmd  by  the  need  to  communicate  in  order  to  or¬ 
ganize  and  structure  the  disparate  data  being  col¬ 
lected.  Tethered  connectivity  for  the  purpose  of 
communication  and  power  may  be  acceptable  in 
visualization  environments  but  could  be  a  large 
barrier  in  surgical  environments.  The  Metaverse 
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project,  for  example,  uses  a  communication  net¬ 
work  to  support  a  grid  of  cameras  that  coopera¬ 
tively  senses  the  state  of  the  display  area  and 
makes  geometric  and  photometric  corrections 
when  required.22  in  that  application,  it  is  of  little 
consequence  that  the  cameras  are  tethered  to  pro¬ 
vide  power  and  network,  since  the  display  area 
can  readily  support  that  infrastructure.  However, 
in  surgical  environments,  where  the  area  of  inter¬ 
est  is  within  the  body,  wireless  transmission  and 
associated  protocols  can  be  very  important.  The 
Massachusetts  Institute  of  Technology’s  Wireless 
Microsensor  System  project^^  emphasizes  the 
study  of  protocols,  networks,  and  designs  for  flex¬ 
ible,  wireless,  and  rapidly  configurable  arrays  of 
sensors. 

Increasingly  powerful,  small,  distributed,  un¬ 
tethered  sensors  have  brought  new  capabilities  and 
imprecedented  flexibility  to  visualization  environ¬ 
ments.  Their  use  also  brings  a  new  set  of  problems 
that  must  be  solved,  including  the  need  for  a  com¬ 
putational  framework  capable  of  fusing  data  from 
the  sensor  network  togedier  with  other  data  rele¬ 
vant  to  the  task.  The  solution  adopted  in  most  vi¬ 
sualization  environments  is  the  distributed  "clus¬ 
ter"  architecture  with  high-speed  communication 
connections  for  data  transfer  and  access. 


Modeling  and  Processing 

The  processing  environment  is  the  glue  that  must 
reconcile  and  fuse  data  from  disparate  sensors  and 
prepare  that  data  for  rendering  and  display.  Fusion 
involves  incorporating  both  sensor  data  and  model 


and  simulation  data  such  as  virtual  models  being 
manipulated  or  data  collected  prior  to  the  real-time 
operation  in  the  environment.  The  modeling  and 
processing  environment  must  transform  data  from 
raw  form  into  the  form  that  will  be  presented  via 
display  interfaces  to  the  viewer. 

V\^en  data  acquired  from  sensors  exactly 
matches  the  end  display  device,  there  is  little  need 
for  a  modeling  and  processing  layer.  The  current  la¬ 
paroscopic  environment  has  been  engineered  so 
that  the  camera  acquires  an  image  sequence  that  is 
mapped  direcdy  onto  the  display  device  (Figure  2, 
left).  Flexibility  and  power  can  be  gained  by  decou¬ 
pling  this  connection  and  inserting  an  intermediate 
distributed  processing  environment,  which  enables 
a  substantid  departure  from  this  sensor-to-display 
direct  mapping  (Figure  2,  right). 

Since  the  early  days  of  simulation  technology,  a 
number  of  strategies,  issues,  and  techniques  have 
been  proposed  to  facilitate  the  development  of 
large-scale  systems  to  support  simulations,^^ 
Surgical  simulation  environments  have  mirrored 
this  trend.  The  Spring  surgical  environment, for 
example,  implements  a  multithreaded  system  for 
managing  a  number  of  sensors  (haptic,  camera) 
while  running  a  tissue  simulator  based  on  the 
mass/spring  physical  simulator. 

Toward  scdable,  massively  parallel  clusters, 
systems  such  as  the  Princeton  Display  WalP®  use  a 
cluster  of  machines  to  run  complex  simulations 
together  with  a  cluster  to  drive  the  distributed 
rendering  environment.  The  simulation  cluster 
produces  simulation  data  in  real-time  and  the  ren¬ 
dering  cluster  renders  it  to  a  scalable,  large-scale 
display  device.  These  parallel,  distributed  systems 


Figure  2:  (left)  The  architecture  minimizes  latency  by  directly  connecting  a  device  (shaded  circle)  driven  by  a  CPU 
to  the  display  device,  (right)  A  scalable,  distributed  architecture  uses  CPU  clusters  on  a  high-speed  network  to  drive 
sets  of  devices,  complex  simulations,  and  a  scalable,  flexible  display  environment. 
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provide  the  required  processing  power  to  run  the 
simulation  and  to  drive  the  large-scale  display. 

In  the  case  of  medical  simulation  and  more 
specifically  laparoscopy,  the  stated  goal26»27  is  to 
build  a  system  that  can  reconcile  imagery  from  ±e 
scope  with  preoperative  imagery  captured  by  CT 
scans  and  MRI,  and  any  other  method  than  might 
be  of  some  service  during  the  laparoscopic  proce- 
dmre.  For  example,  comparison  or  recognition  of 
anatomy  with  respect  to  a  large  database  of  proce¬ 
dures  might  be  helpful.  It  is  clear  that  ±ese  goals 
cannot  be  accomplished  with  a  computing  environ¬ 
ment  that  so  tightly  couples  the  data  acquisition  de¬ 
vices  (camera)  with  the  display  device  (monitor). 
Parallel  distributed  computing  environments  may 
be  the  answer,  and  it  may  be  the  case  that  surgical 
simulation  and  eventually  operating  rooms  become 
driven  by  massively  parallel  clusters  of  computers 
designed  specifically  to  manage  distributed  sensors 
and  preoperative,  potentially  collaborative  patient 
databases. 

The  tdtimate  end  of  the  visualization  environ¬ 
ment  is  the  presentation  of  the  data  to  the  viewer, 
who  then  interprets  the  data  and  reacts,  explores, 
and  plans  accordingly.  Display  is  normally  linked  to 
the  CRT,  and  the  tight  coupling  that  exists  between 
the  scope  and  the  CRT  reduces  the  flexibility  that 
otherwise  might  be  achieved.  Visualization  envi¬ 
ronments  have  moved  toward  a  decoupling  of  indi¬ 
vidual  sensors  and  simulation  elements  from  the 
end  display  environment,  providing  flexibility  in 
how  to  integrate  and  present  information  to  the 
viewer. 


Rendering  and  Display 

Although  visualization  environments  emphasize  vis¬ 
ible,  image-driven  display,  they  can  also  incorpo¬ 
rate  haptic  and  vestibular  manipulations  and  audio 
rendering  to  induce  spatially  localized  effects.  High- 
fidelity  multisensory  display  is  desirable  in  order  to 
increase  bandwidth  at  the  human-machine  inter¬ 
face.  It  is  implausible  to  expect  to  increase  the 
amount  of  data  in  the  system  through  large  num¬ 
bers  of  sensors  and  simulation  elements  while  ex¬ 
pecting  the  observer  to  continue  to  imderstand  and 
interpret  that  new  data  without  difficulty. 

Along  with  a  desire  to  support  multisensory  dis- 
playSy  the  need  for  flexibility  has  also  become  ap¬ 
parent.  The  desire  for  flexibility  in  the  display  envi¬ 
ronment  runs  counter  to  the  rigidity  of  most  visual¬ 
ization  tasks.  Commonly  available  display  devices, 
such  as  CRTs,  stereographic  systems,  the  Cave 


Automatic  Virtual  Environment  (CAVE),^^  the 
Reconfigurable  Advanced  Visualization  Environment 
(RAVE),2^  head-mounted  displays,  and  other  similar 
systems,  are  used  to  display  processed  data,  3D  rep¬ 
resentations,  and  simulated  environments.  However, 
these  systems  are  all  notoriously  inflexible,  and  many 
are  monolithic  and  expensive. 

How  can  display  systems  support  the  need  for 
scalability,  multisensory  input,  and  flexibility?  First, 
they  leverage  the  distributed  computing  architec¬ 
ture  (Figure  2).  Second,  they  incorporate  a  special 
set  of  sensors  that  monitor  the  environment  itself 
in  order  to  provide  flexibility.  Thus,  the  distributed 
computing  environment  is  responsible  for  control¬ 
ling  and  sampling  sensor  input,  running  simulation 
code,  controlling  sensors  that  monitor  the  visual¬ 
ization  environment  itself,  and  combining  every¬ 
thing  into  a  form  that  can  be  rendered  on  the  local 
display  environment,  which  may  itself  be  flexibly 
changing. 

A  new  degree  of  flexibility  can  now  be  achieved 
through  sensor  monitoring.  In  the  display  realm, 
this  has  been  demonstrated  through  camera-based 
approaches  for  geometric  and  photometric  blend- 
ijig  22,3o.3ipiguf  es  3  and  4  show  how  a  four-projector 
system  that  is  casually  aligned  (the  projected  im¬ 
ages  do  not  line  up  correctly  on  the  display  surface) 
can  be  corrected  automatically.  The  cameras  detect 
the  geometry  and  compare  it  to  what  should  be  vis¬ 
ible,  since  they  can  be  told  what  to  expect  to  see 
(Figure  3).  Once  the  geometric  correction  is  calcu¬ 
lated,  it  can  be  applied  in  order  to  form  a  unified 
display  area  across  the  projection  surface.  Figure  4 
Oeft)  shows  a  grid  after  the  correction  is  applied. 
The  grid  aligns  perfectly  even  though  the  underly¬ 
ing  projectors  do  not  physically  align. 

Illumination  compensation  applied  to  areas 
where  more  than  one  projector  overlaps  makes 
the  whole  illuminated  area  appear  to  be  a  single 
.image.  The  geometric  and  photometric  correc¬ 
tion,  applied  on-the-fly  from  cameras  that  ob¬ 
serve  how  the  projectors  are  configured,  creates 
the  appearance  of  a  unified,  seamless  display 
that  is  very  flexible  and  scalable  in  resolution 
and  size.  Figure  4  (right)  shows  an  observer 
viewing  volumetric  data  set  projected  onto  a 
four-projector  display.  The  contributing  projec¬ 
tors  are  so  well  calibrated  that  it  is  very  difficult 
to  tell  which  projector  is  illuminating  particular 
pixels  on  the  wall. 

This  kind  of  flexibility  allows  projectors  to  be 
rapidly  configured  to  cover  a  small  amoimt  of  area 
at  a  very  high  number  of  dots  per  inch  for  applica¬ 
tions  that  require  dose-in,  very  bright  work.  On  the 


T6 


Seales  and  Caban 


Figure  3:  Projection-based  display  hardware  can  be  casually  aligned  (left)  and  calibrated  via  automatic,  camera- 
based  algorithms  (right),  which  calculate  and  apply  the  correct  alignment  transformation. 


other  hand,  projectors  can  also  be  adjusted  to  cover 
large  areas  to  fill  entire  walls  with  imagery.  A  blend 
is  also  possible:  a  high-resolution  inset  can  be  over¬ 
laid  in  an  area  where  more  detail  may  be  required. 
In  any  case,  the  cameras  detect  the  geometry  and 
correct  the  imagery  as  required. 

In  the  case  of  the  surgical  simulation  environ¬ 
ment,  dedicated  sensors  can  be  designed  to  calcu¬ 
late  the  transformation  between  real-time  scope  po¬ 
sition  (perhaps  given  by  a  tracking  sensor)  and  a 
set  of  preoperative  data  such  as  an  MRI  or  a  CT 
scan.  likewise,  flexible,  scalable  display  technology 
may  help  to  achieve  a  much  higher  degree  of  flexi¬ 
bility  and  configurability  for  particularly  imusual 
procedures  and  patient  positions. 

The  key  trends  in  visualization  of  advanced 
hardware  capability  and  distributed  computing  en¬ 
vironments  have  led  to  advances  in  data  acquisition, 
processing  and  modeling,  and  rending  and  display. 
The  distributed  computing  framework  can  incorpo¬ 


rate  a  large  number  of  independent,  powerful  sen¬ 
sors  and  simulation  elements  and  helps  to  decou¬ 
ple  the  sensors  from  the  end  display  technology, 
leading  to  flexibility  and  new  algorithms  for  fusing 
and  rendering  disparate  data.  However,  a  number 
of  issues  must  be  addressed  when  these  techniques 
are  applied  in  the  laparoscopic  arena. 


Visualization  and  Laparoscopy:  Issues 

Non-mission-critical  visualization  tasks,  such  as 
scholarly  study  and  the  exploration  of  digitized 
sculpture,  or  the  analysis  of  weather  patterns  or 
fluid  flow,  can  all  be  accomplished  as  a  casual,  re¬ 
peatable,  non-time-critical  study  in  an  environment 
that  need  not  be  fault  tolerant.  The  relative  lack  of 
constraints  has  freed  visualization  researchers  to  ex¬ 
plore  new  architectures  and  approaches,  and  much 


Figure  4:  After  calibration  the  projectors  work  cooperatively  to  display  a  seamless  image. 
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of  what  has  been  developed  is  indeed  intriguing. 
The  requirements  for  medical  analysis  are  much 
higher  than  the  casual  scientific  visualization  task, 
however.  A  few  such  requirements  are: 

•  Equipment  must  be  nondistracting  and  comfort¬ 
able  for  the  surgeon 

•  Latency  must  be  minimal 

•  Color  fidelity  is  crucial  for  identifying  and  as¬ 
sessing  anatomy 

•  Spatial  fidelity  is  crucial 

•  Calibration  and  registration  tolerances,  which 
can  never  be  zero,  must  be  very  small 

Equipment:  Certain  visualization  devices  are 
very  difficult  to  apply  in  the  medical  realm,  because 
they  can  be  unconrfortable.  Head-mounted  displays 
and  data  gloves,  for  example,  can  be  cumbersome 
and  difficult  to  manage  in  a  surgical  setting.  The  re¬ 
quirement  that  the  equipment  be  sterile,  nondis¬ 
tracting,  and  capable  of  use  for  long  periods  of  time 
without  adverse  effects,  such  as  sickness  and  dis¬ 
orientation,  makes  it  difficult  to  incorporate  these 
kinds  of  visualization  devices  direcdy  into  surgical 
applications. 

Latency:  Latency  is  the  time  difference  be¬ 
tween  the  acquisition  of  data  at  die  sensor  and  the 
display  of  that  data  to  the  observer.  The  end-to-end 
latency  present  in  a  system  is  obvious  when  sensor 
data  (the  image)  and  direct  observation  (cues  from 
probes)  are  juxtaposed.  Unfortunately,  any  pro¬ 
cessing  between  acquisition  and  display  adds  laten¬ 
cy.  The  current  system  for  laparoscopy  reduces  la¬ 
tency  to  a  very  low  level  by  mapping  the  camera 
sign^  directly  to  the  CRT.  Users  will  likely  criticize 
any  system  that  creates  a  latency  that  is  higher  than 
the  existing  system,  despite  any  advantages  it  may 
provide  in  such  things  as  flexibility  and  enhance¬ 
ment.  The  latency  constraint  is  a  perfect  example 
of  how  difficult  it  is  to  apply  advances  in  visualiza¬ 
tion  technology  to  laparoscopy.  Laparoscopic  sys¬ 
tems  minimize  latency,  and  it  will  be  difficult  to 
move  to  a  parallel,  distributed  system  without  sac¬ 
rificing  something  firom  this  constraint.  One  can 
argue  that  it  would  be  easier  to  tolerate  more  la¬ 
tency  than  to  work  in  a  3D  space  using  only  a 
monocular  2D  image  sequence.  But  for  surgeons 
who  are  already  accomplished  at  using  the  2D,  low- 
latency  imagery  to  get  ±e  job  done,  a  higher-laten¬ 
cy  system  would  involve  loss  of  skill  and  the  need 
for  retraining. 

Color  fidelity:  Digital  cameras  are  built  to 
very  high  standards  for  laparoscopy.  The  benefit 
of  the  high  fidelity  is  lost,  however,  if  this  signal  is 


displayed  on  a  device  that  inadequately  reproduces 
color.  Unfortunately,  many  display  devices  do  not 
match  the  high  standards  of  the  camera.  In  particu¬ 
lar,  commodity  display  devices  (CRTs,  LCD  panels, 
and  projectors)  often  trade  color  fidelity  for  a  vari¬ 
ety  of  other  characteristics,  such  as  brighmess  and 
portability.  In  particular,  the  commodity  projectors 
that  are  used  for  cooperative,  scalable  display  in  vi¬ 
sualization  environments  often  show  large  varia¬ 
tions  in  color  and  brighmess  characteristics. 

Spatial  fidelity:  Mismatch  between  acquisi¬ 
tion  and  display  means  that  algorithms  must  inte¬ 
grate  and  sometimes  downsample  data.  The  algo¬ 
rithms  for  blending  and  fusing  data  can  cause  arti¬ 
facts  that  mar  spatial  fidelity.  The  display  has  often 
been  a  limiting  factor  in  many  applications,  which  is 
why  scalable  display  technology  may  be  a  valuable 
answer  for  expanding  display  devices  to  handle  bet¬ 
ter  camera  resolution  and  to  present  integrated  data 
from  other  devices  in  close  proximity  to  camera 
data. 

Calibration  and  registration:  Single  de¬ 
vice  calibration  can  often  be  done  at  the  factory  to 
specified  tolerances.  The  distributed  architecmre 
shown  in  Figure  2  elevates  the  need  for  real-time, 
consistent,  accurate  calibration  and  registration  be¬ 
tween  devices  in  a  way  that  is,  however,  difficult  or 
impossible  to  do  off-line.  Cooperative  device  com¬ 
munication  and  calibration  is  essential  in  the  dis¬ 
tributed  enviromnent. 

Given  these  requirements  and  the  high  degree 
of  reliability  required  of  the  solutions,  it  remains 
challenging  to  apply  visualization  technology  in  the 
domain  of  laparoscopy.  Clearly,  simulation  envi¬ 
ronments  are  the  first  step  for  development,  de¬ 
ployment,  and  testing.  The  double  barrier  of  the  op¬ 
erating  room  and  the  minimally  invasive  procedure 
stand  as  one  of  the  most  challenging  environments 
in  which  to  apply  new  visualization  technology, 
with  a  potentid  payoff  that  is  very  high.  In  order  to 
achieve  a  substantial  breakthrough,  it  is  crucial  to 
embrace  and  e:q)loit  the  trends  that  are  driving  vi¬ 
sualization  technology  and  to  find  ways  to  address 
the  associated  challenges  ±at  are  unique  to  mini¬ 
mally  invasive  surgery. 
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